Monocular motion stereo-based free parking space detection apparatus and method

ABSTRACT

A monocular motion stereo-based automatic free parking space detection system is disclosed. The system acquires image sequences with a single rearview fisheye camera, three-dimensionally reconstructs the vehicle rearview by using point correspondences, and recovers metric information from a known camera height to estimate the positions of adjacent vehicles thereby detecting the free parking spaces. By using de-rotation-based feature selection and 3D structure mosaicking the degradation of the 3D structure near the epipole is solved and it is not necessary to use the unreliable odometry due to its accuracy depending on road conditions.

CROSS-REFERENCE TO RELATED APPLICATION

This non-provisional application claims priority under 35 U.S.C §119(a)on Patent Application No. 10-2008-0028581 filed in Korea on Mar. 27,2008, the entire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to a free parking detection apparatus andmethod. More particularly, the present invention relates to a monocularmotion stereo-based free parking space detection apparatus and methodfor capturing the automobile rearview image and detecting from the imagea free parking space.

BACKGROUND OF THE DISCLOSURE

An automatic parking system provides convenience for drivers byautomatically finding free parking spaces and steering their automobilestoward them. Recently, there have been increased interests in automaticparking systems. By the customers' interests and the success stories ofseveral automatic parking systems, many car manufacturers and componentmanufacturers are preparing to release self-parking products.

Automatic parking system systems consist of three components: pathplanning (including free parking space detection), an automatic steeringand braking system used to implement the planned trajectory, and the HMI(Human Machine Interface), which can be used to receive driver's inputand provide visual information of the ongoing parking process.

Free parking space detection has been implemented by using variousmethods: the ultrasonic sensor-based method, the laser scanner-basedmethod, the short range radar network-based method, and the vision-basedmethod. Among these, the vision-based method has proven the mostattractive to drivers because it visualizes parking situations andperforms procedures to make drivers feel safer. The vision-based methodcan be categorized into four approaches: the parking space marking-basedapproach, the binocular stereo-based approach, the light stripeprojection-based approach, and the monocular motion stereo andodometry-based approach.

The first approach recognizes parking space markings. Xu et al.developed color vision-based localization of parking spaces. This methoduses color segmentation based on RCE neural networks, contour extractionbased on the least square method, and inverse perspectivetransformation. Jung et al. proposed the semi-automatic parking assistsystem which recognized marking lines by using the Hough transform in abird's eye view edge image captured with a wide-angle camera. In thisway, target spaces can be detected with a single image at a relativelylow computational cost. Also, a general configuration of a rearviewcamera (a single fisheye camera) can be used. However, it cannot be usedwhen parking space markings are not visible. Also, performance can bedegraded by poor visual conditions such as stains, shadows or occlusionby adjacent vehicles.

The second approach recognizes adjacent vehicles by using a binocularstereo-based 3D reconstruction method. Kaempchen et al. developed theparking space estimation system which uses a feature-based stereoalgorithm, a template matching algorithm on a depth map and a 3D fittingto the 2D planar surface model of the vehicle. This approach can easilyrecover metric information from the fixed length of the baseline and thecamera's extrinsic parameters need not be estimated every time. However,this requires extra costs and space for the equipment because a stereocamera is not a general configuration of a rearview camera. Sub-pixelaccuracy is required when there are short—baselines between the twocameras, and point correspondences are difficult to find when there arewide baselines.

Jung et al. developed a method which combines the parking spacemarking-based approach and the binocular stereo-based approach. Theseresearchers used obstacle depth maps for establishing the search rangeand simple template matching for finding the exact location of freeparking spaces. This method is robust to noise factors such as stains,trash and shadows when compared to the parking space marking-basedmethod, but it can be only used when both obstacle depth and parkingspace markings are available.

The third approach recognizes adjacent vehicles by using a laserprojector and a single rearview camera. Jung et al. developed a methodwhich identified free parking spaces by analyzing the light stripe (onbackward objects) produced by the laser projector. This approach can beapplied to dark underground parking lots and the algorithm for acquiring3D information is relatively simple. A general configuration of arearview camera can be used. However, this approach cannot be usedduring the day due to the presence of sunlight.

The fourth approach recognizes adjacent vehicles by using a monocularmotion stereo method and odometry. Fintzel et al. proposed a systemwhich provides a rendered image from a virtual viewpoint for betterunderstanding of parking situations and procedures. This system obtainsexternal parameters and metric information from odometry andreconstructs the 3D structure of the parking space by using pointcorrespondences. However, these researchers did not present a freeparking space detection method. A general configuration of the rearviewcamera can be used. However, odometry information can be erroneous whenroad conditions are slippery due to rain or snow.

DISCLOSURE OF THE INVENTION

In order to solve the above-identified problems, the present disclosureproposes to acquire an image sequence by capturing the images of freeparking spaces with an automobile rearview image captor or camera,three-dimensionally reconstruct view behind the automobile by usingpoint correspondences on the image sequence, and recover metricinformation on the 3D structures from the known camera height and detectfree parking spaces by estimating the positions of adjacent vehicles.

An embodiment of the present disclosure provides a vehicle free parkingspace detection apparatus comprising a feature points tracker fortracking feature points by receiving input images from an image captorand acquiring point correspondences through analyzing the input images;a three-dimensional structure generator for generating athree-dimensional structure through three-dimensionally reconstructingthe free parking space by using the feature points; a three-dimensionalmosaic structure generator for generating a three-dimensional structuremosaic through reselecting the feature points to reconstruct thethree-dimensional structure by using a de-rotation-based featureselection and mosaic the reconstructed three-dimensional structure byestimating a similarity transformation; a metric recoverer for acquiringmetric information from the three-dimensional structure mosaic by usingan image captor height ratio; and a free parking space detector fordetecting the free parking space by estimating the positions of adjacentvehicles in the three-dimensional structure mosaic.

Another embodiment of the present disclosure provides a vehicle freeparking space detection apparatus comprising: an image captor forcapturing the free parking space to generate input images and deliverthe same; a user interface for displaying the detected free parkingspace in an output image; a free parking space detector for receivingthe input images to track feature points by acquiring pointcorrespondences through analyzing the input images, generating athree-dimensional structure through three-dimensionally reconstructingthe free parking space by using the feature points, generating athree-dimensional structure mosaic through reselecting the featurepoints to reconstruct the three-dimensional structure by using ade-rotation-based feature selection and mosaic the reconstructedthree-dimensional structure by estimating a similarity transformation,acquiring metric information from the three-dimensional structure mosaicby using an image captor height ratio, and detecting the free parkingspace by estimating the positions of adjacent vehicles in thethree-dimensional structure mosaic to generate the detected free parkingspace and deliver the same; a sensor having a number of sensing unitsfor recognizing driving conditions of the vehicle to generate anddeliver vehicle driving condition information; a parking assistancecontroller for receiving the driving condition information from thesensor to estimate the vehicle position, generating a path plan to parkthe vehicle into the free parking space by using the detected freeparking space delivered, and generating and delivering control signalsto execute the parking into the free parking space; an active steeringsystem for steering the vehicle in response to the control signals upontheir receipt; and an active breaking system for breaking the vehicle inresponse to the control signals upon their receipt.

Yet another embodiment of the present disclosure provides a method for avehicle free parking space detection apparatus to detect the freeparking space comprising the steps of tracking feature points byreceiving input images from an image captor and acquiring pointcorrespondences through analyzing the input images; generating athree-dimensional structure through three-dimensionally reconstructingthe free parking space by using the feature points; reconstructing thethree-dimensional structure through reselecting the feature points byusing a de-rotation-based feature selection; generating athree-dimensional structure mosaic through mosaicking the reconstructedthree-dimensional structure by estimating a similarity transformation;acquiring metric information from the three-dimensional structure mosaicby using an image captor height ratio; and detecting the free parkingspace by estimating the positions of adjacent vehicles in thethree-dimensional structure mosaic.

As described above, the present disclosure can solve the degradation ofthree-dimensional structures near the epipole by using de-rotation-basedfeature selection and three-dimensional structure mosaicking.

In addition, the present disclosure suggests an efficient way oflocating free parking spaces in the three-dimensional point clouds.

Also, the unreliable odometry may not be used in accordance with thepresent disclosure because its accuracy depends largely on roadconditions through estimating the external parameters by using only theimage information and recovering the metric information from the ratioof the camera height in the reconstructed world and the real world.

This may implement a monocular motion stereo-based free parking spacedetection system.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentdisclosure will be more apparent from the following detailed descriptiontaken in conjunction with the accompanying drawings, in which:

FIG. 1 is a schematic drawing of an automatic parking system using amonocular motion stereo-based free parking space detection apparatusaccording to an embodiment of the present invention;

FIG. 2 is a block diagram of a monocular motion stereo-based freeparking space detection method according to an embodiment of the presentinvention;

FIG. 3 is a drawing for showing an epipole location in a typical parkingsituation;

FIGS. 4A, B show the location of the epipole in the last frame of theimage sequence and the reconstructed rearview structure;

FIGS. 5A, B show de-rotation-based feature selection procedure;

FIGS. 6Aa-6Ac, 6Ba-6Bc, 6Ca-6Cc show the reconstructed 3D structureswith and without the proposed feature selection;

FIGS. 7A, B show the 3D structures when using and without using thefeature selection and 3D mosaicking method, respectively;

FIG. 8 is a schematic diagram showing a configuration of rearviewcamera;

FIG. 9 is a graph showing a density of the Y-axis coordinate of the 3Dpoints;

FIGS. 10A, B show the result of metric recovery in camera-view andtop-view, respectively;

FIGS. 11A, B show the dimension reduction result and the isolated pointdeletion result, respectively;

FIGS. 12A, B show the outline point selection;

FIGS. 13A, B show a search range of the other adjacent vehicle and freeparking space localization;

FIG. 14 shows the final result of free parking space detection depictedon the last frame of the image sequence;

FIG. 15 is a photograph of a fisheye camera and laser scanner mounted onthe automobile;

FIGS. 16Aa-16Ab, 16Ba-16Bb, 16Ca-16Cb show comparison of thereconstructed rearview structures when using and without using theproposed method;

FIGS. 17Aa-17Ac, 17Ba-17Bc, 17Ca-17Cc, 17Da-17Dc show six successfuldetections;

FIGS. 18A, B, C, D show four types of failures; and

FIGS. 19A, B show evaluation results by using laser scanner data.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

Hereinafter, an exemplary embodiment of the present disclosure will bedescribed with reference to the accompanying drawings. In the followingdescription, the same elements will be designated by the same referencenumerals although they are shown in different drawings.

Further, in the following description of the present disclosure, adetailed description of known functions and configurations incorporatedherein will be omitted when it may make the subject matter of thepresent disclosure rather unclear.

FIG. 1 is a schematic drawing of an automatic parking system 100 using amonocular motion stereo-based free parking space detection apparatusaccording to an embodiment of the present invention.

The automatic parking system 100 includes an image captor 110, userinterface 120, free parking space detector 130, parking assistancecontroller 140, sensor 150, active steering system 160 and activebreaking system 170.

The image captor 110 is a means mounted on a vehicle to capture images.It may be mounted to various locations of the vehicle such as its front,rear or lateral sides so that it may capture the image of a free parkingspace to generate an input image for delivery to user interface 120,free parking space detector 130, parking assistance controller 140 andothers. For this purpose, image captor 110 may be implemented by variousimage capturing apparatuses including a film camera, digital camera,digital camcorder, CCTV, etc.

The user interface 120 is a means mounted on the vehicle to input/outputdata or commands. It may be mounted on the vehicle near its interior,driving seat or passenger seat to receive driver's data or commandsinput and deliver the same to free parking space detector 130 andparking assistance controller 140, and receive and output data from andto image captor 110, free parking space detector 130 and parkingassistance controller 140.

Additionally, upon receiving the input image from image captor 110 theuser interface 120 displays and transfers the same to free parking spacedetector 130. For this purpose, user interface 120 may be implemented byvarious input/output devices such as a liquid crystal display or LCD,touch pad, keyboard, mouse and/or touch screen.

The free parking space detector 130 is a means for executing datacommunications and calculations which upon receipt of the input imagefrom image captor 110 detects the free parking space in the input imageand generate images indicative of the detected free parking space as anoutput image for transferring to user interface 120, which is thencontrolled to display the output image.

To this end, the free parking space detector 130 may comprise a featurepoint tracker 132, a three-dimensional structure generator 134, athree-dimensional mosaic structure generator 136, metric recoverer 138and space detector 139.

The feature point tracker 132 tracks feature points by receiving theinput images from image captor 110 and acquiring point correspondencesthrough analyzing the input images. Here, feature point tracker detectsthe feature points through analyzing the input images and tracks thefeature points through the input images by using the Lucas-Kanademethod.

The three-dimensional structure generator 134 generates athree-dimensional structure through three-dimensionally reconstructingthe free parking space by using the feature points. Here,three-dimensional structure generator 134 selects key frames of theinput images, estimates motion parameters and then calculates threedimensional coordinates of the feature points by using a lineartriangulation method to generate the three-dimensional structure.

The three-dimensional mosaic structure generator 136 generates athree-dimensional structure mosaic through reselecting the featurepoints to reconstruct the three-dimensional structure by using ade-rotation-based feature selection and mosaic the reconstructedthree-dimensional structure by estimating a similarity transformation.

The metric recoverer 138 acquires metric information from thethree-dimensional structure mosaic by using an image captor heightratio. Here, metric recoverer 138 acquires the image captor height ratioby determining the image captor height in the three dimensional mosaicstructure through a tilting angle compensation, a densityestimation-based ground plane detection, and a three-dimensional planeestimation-based ground plane refinement in its location andorientation.

The space detector 139 detects the free parking space by estimating thepositions of adjacent vehicles in the three-dimensional structuremosaic. Here, the space detector 139 reduces the three dimensionalstructure mosaic with a top-view, deletes isolated points fromneighbors, acquires outline points of the adjacent vehicles with anincoming angle and the distance from the image captor center andestimate a corner point and orientation of the adjacent vehicle, wherebydetecting the free parking space.

The space detector 139 additionally estimates other adjacent vehicleslocated opposite the estimated vehicle by setting a circle with apredetermined radius as the search range of the other adjacent vehicleswith the center of the circle being located a predetermined distancefrom the corner point in the orthogonal direction of the orientation.

Here, the free parking space detector 130 and its components of featurepoint tracker 132, three-dimensional structure generator 134,three-dimensional mosaic structure generator 136, metric recoverer 138and space detector 139 may be implemented by an independent apparatushaving memories for storing programs to perform the correspondingfunctions and microprocessors for executing the stored programs in thememories although their implement may be possible through softwaremodules which are run by hardware (memories, microprocessors and thelike) provided in automatic parking system 100.

A parking assistance controller 140 receives driving conditioninformation from a sensor 150 to estimate the vehicle position, receivesthe detected free parking space from free parking space detector 130 andgenerate a path plan to park the vehicle into the free parking space,and generates control signals to execute the parking into the freeparking space for transmitting the same signals to active steeringsystem 160 and active breaking system 170.

Also, the parking assistance controller 140 may control the generatedpath plan to be transmitted to user interface 120 that displays the pathplan as it controls the images of the parking process of the vehicleinto the free parking space to be captured by image captor 110 andtransmitted to user interface 120, which displays the captured images.

For this purpose, parking assistance controller 140 comprises a pathplan generator 142 for generating a path plan to park the vehicle byusing the detected free parking space delivered from free parking spacedetector 130, a path tracking controller 144 for tracking the set pathplan based on the vehicle position and driving conditions to generatecontrol signals for controlling the parking of the vehicle into the freeparking space and transmit the control signals to active steering system160 and active breaking system 170, and a vehicle position estimator 146for estimating the vehicle position by using the driving conditioninformation received from sensor 150.

Here, the parking assistance controller 140 and its components of pathplan generator 142, path tracking controller 144 and vehicle positionestimator 146 may be implemented by an independent apparatus havingmemories for storing programs to perform the corresponding functions andmicroprocessors for executing the stored programs in the memoriesalthough their implement may be possible through software modules whichare run by hardware (memories, microprocessors and the like) provided inautomatic parking system 100.

Sensor 150 may be provided with a number of sensor units for recognizingdriving conditions of the vehicle, i.e. a wheel speed sensor, a steeringangle sensor, a yaw rate sensor, acceleration sensor, etc. to sense thevehicle driving conditions and generate their equivalent electric sensorsignals for vehicle driving condition information, which is to betransmitted to vehicle position estimator 146 of parking assistancecontroller 140.

The active steering system 160 is a steering assistance device forinducing a safe steering of the vehicle by using various sensors andcontrol devices from driver steering inputs which steers the vehicle inresponse to the control signals received from parking assistancecontroller 140.

Although not limiting, active steering system 160 may encompass anelectronic power steering system (EPS), a motor driven power steeringsystem (MDPS), an active front steering system (AFS), etc.

The active breaking system 170 is to restrict the vehicle movement speedthrough changing the degree of breaking the vehicle in response to thecontrol signal received from parking assistance controller 140 therebyhalting the vehicle.

In order to achieve this, active breaking system 170 may encompass ananti-lock brake system (ABS), an electronic stability control system(ESC), etc. among others.

FIG. 2 is a block diagram of a monocular motion stereo-based freeparking space detection method according to an embodiment of the presentdisclosure.

In the disclosed monocular motion stereo-based free parking spacedetection method, automatic parking system 100 in step 210 receives thecaptured image input of free parking spaces from image captor 110,tracks feature points of the image input in step 230, 3D-reconstructsthe parking structure using the feature points in step 230, selects thefeature points using the de-rotation-based feature selection in step240, reconstructs 3D structures by using the selected pointcorrespondences and forms the reconstructed 3D structure into a 3Dstructure mosaic by estimating the similarity transformation in step250, recovers 3D structure by using the camera height ratio in step 260,and detects free parking spaces by estimating the positions of adjacentvehicles in step 270. Then, automatic parking system 100 may generatethe image representing the detected free parking space as output imageto transmit the same to be displayed on user interface 120 in step 280.

In the present disclosure, the rearview structures arethree-dimensionally reconstructed by using a single fisheye camera andfree parking spaces are found in the 3D structure through the stages ofvideo (image sequence) acquisition, feature point tracking, 3Dreconstruction, 3D structure mosaicking, metric recovery, and freeparking space detection.

Compared to prior arts, the disclosed system provides three importantadvantages. First, the degradation of the 3D structure near the epipoleis solved by using de-rotation-based feature selection and 3D structuremosaicking. This is a serious problem when reconstructing 3D structureswith an automobile rearview camera because the epipole is usuallylocated on the image of an adjacent vehicle which must be preciselyreconstructed. Second, an efficient method for detecting free parkingspaces in 3D point clouds is provided. Advantageously, odometry is notused for its unreliability, since the accuracy would have depended onroad conditions. The external parameters may be estimated by using onlythe image information. The metric information may be recovered from theratio of the camera heights in the reconstructed world and the realworld.

Hereinafter, the present disclosure will be explained in detail inSections 1 to 4. Section 1 discusses the methods for pointcorrespondences and 3D reconstruction. Section 2 explains the problem ofthe epipole and offers a solution. Section 3 describes metric recoveryand the free parking space detection procedure. Section 4 presents theexperimental results in comparison with the laser scanner data.Incidentally, for the purpose of easy comprehension image captor 110will be referred to as ‘camera’.

1. Monocular Motion Stereo-Based 3D Reconstruction

1.1 Point Correspondences

Point correspondences in two different images acquired with a movingcamera have to be found in order to estimate the motion parameters and3D structures. In this disclosure, three different approaches arecompared. The first approach finds a small number of reliable pointcorrespondences to estimate the fundamental matrix and matches featurepoints by using the epipolar constraints on the rectified images. Thisis called guided matching, which requires the fisheye images to beundistorted and rectified.

The second approach finds point correspondences by using a naivealgorithm, and rejects false matches by using an outlier rejectionmethod designed for cameras on intelligent vehicles. Even though thisapproach is fast and produces few mismatches, it is difficult to findpoint correspondences on automobile surfaces due to the lack of featureson these surfaces or featurelessness.

The third approach detects feature points and tracks them through imagesequences. Since this method tracks the feature points betweenconsecutive images, it can find many point correspondences on automobilesurfaces. However, computational costs are high because the algorithmhas to be applied to many images.

The first and second approaches require at least two images. The memorysize for saving the images is small but it is difficult to select keyframes without saving the whole sequence. The third approach alsorequires at least two images every moment if it is implemented inreal-time by using Field-Programmable Gate Array (FPGA). Since the pointcorrespondences are saved for each frame, key frames can be selected byusing the tracking results. Considering this comparison, the thirdapproach of tracking method is preferred in the present disclosure.

For tracking, the present disclosure chooses the Lucas-Kanade methodbecause it produces accurate results, offers affordable computationalpower, and there are some existing examples of real-time hardwareimplementations. This method also uses the least square solution ofoptical flows. If I and J are two consecutive images and x and Ω denotethe feature position and the small spatial neighborhood of x,respectively, then the goal is to find the optical flow vector, v whichminimizes:

$\begin{matrix}{\min\limits_{v}{\sum\limits_{x = \Omega}\left\{ {{I(x)} - {J\left( {x + v} \right)}} \right\}^{2}}} & (1)\end{matrix}$The solution of Eq. (1), v_(opt) is given by:

$\begin{matrix}{{v_{opt} = {G^{- 1}b}}{{G = {\sum\limits_{x \in \Omega}\begin{bmatrix}I_{x}^{2} & {I_{x}I_{y}} \\{I_{x}I_{y}} & I_{y}^{2}\end{bmatrix}}},{b = {\sum\limits_{x \in \Omega}\begin{bmatrix}{\delta\; I} & I_{x} \\{\delta\; I} & I_{y}\end{bmatrix}}}}} & (2)\end{matrix}$I_(x) and I_(y) are the image gradients in the horizontal and verticaldirections, respectively, and δI is the image pixel difference. Sincethe matrix G is required to be non-singular, the image location wherethe minimum eigenvalue of G is larger than the threshold may be selectedas a feature point and tracked through the image sequence.1.2.3 3D Reconstruction

Once the point correspondences are obtained, the structure of theparking space may be three-dimensionally reconstructed by using thefollowing three steps: key frame selection, motion parameter estimation,and triangulation. For the purpose of 3D reconstruction, the key frameswhich determine the 3D reconstruction interval must be appropriatelyselected. If there is not enough camera motion between the two frames,the computation of the fundamental matrix may be inaccurate and in theopposite case, the number of corresponding points may be decreased.

The present disclosure uses a simple but less general method which usesthe average length of optical flow. This method works well becauserotational motion is always induced by translational motion inautomobile rearview cameras. Since parking spaces should bereconstructed at the driver's request, the last frame may be selected asthe first key frame.

The second key frame may be selected when the average length of opticalflow from the first key frame exceeds the threshold. The next key framemay be selected in the same way. The threshold value may be set to 50pixels and this makes the baseline length approximately 100˜150 cm.

Once the key frames are selected, the fundamental matrix is estimated toextract the relative rotation and translation values between the twocameras. For this task, the present disclosure uses the Random SampleConsensus (hereinafter referred to as RANSAC) followed by theM-Estimator. Torr and Murray found this to be an empirically optimalcombination. Also, with automobile rearview images experiments wereperformed using the various methods suggested by Armangue, X., Salvi,J.: Overall view regarding fundamental matrix estimation. Image VisionComputing 21(2), 205-220 (2003).

The RANSAC is based on randomly selecting a set of points to compute thecandidates of the fundamental matrix by using a linear method. Thismethod calculates the number of inliers for each fundamental matrix andchooses the one which maximizes it. In the experiments, once thefundamental matrix was determined, it was refined by using all theinliers. The M-Estimator may be used to reduce the effect of theoutliers by measuring the residual value of each point correspondence.It is considered that r_(i) is the residual value of x′_(i) ^(T),Fx_(i), where x′_(i) and x_(i) are the coordinates of the pointcorrespondences in two images and F is the fundamental matrix. Then, theM-Estimators may be based on solving Eq. (3):

$\begin{matrix}{\min\limits_{F}{\sum\limits_{i}{w_{i}\left( {x_{i}^{\prime\; T}{Fx}_{i}} \right)}^{2}}} & (3)\end{matrix}$in which w_(i) is a weight function and Huber's function is used in Eq.(4):

$\begin{matrix}{w_{i} = \left\{ \begin{matrix}1 & {{r_{i}} \leq \sigma} \\{\sigma/{r_{i}}} & {\sigma < {r_{i}} \leq {3\sigma}} \\0 & {{3\sigma} < {r_{i}}}\end{matrix} \right.} & (4)\end{matrix}$In order to obtain σ, the robust standard deviation in Eq. (5) may beused:σ=1.4826{1+5/(n−7)}median_(i) |r _(i)|  (5)

The essential matrix may be calculated by using the fundamental matrixand the camera intrinsic parameters matrix (K). The camera intrinsicparameters are pre-calibrated because they do not change in the presentdisclosure. The four combinations of the rotation matrix (R) and thetranslation vector (t) may be extracted from the essential matrix. Sinceonly the correct combination may allow the 3D points to be located infront of both cameras, randomly selected several points arereconstructed to determine the correct combination. After that, the 3Dcoordinates of each point correspondence may be calculated by using alinear triangulation method. If P and P′ represented the projectionmatrices of the two cameras and X represented the 3D coordinate of thepoint correspondences, Eq. (6) would have appeared as follows:x×(PX)=0x′×(P′X)=0  (6)

By combining the above two equations into the form AX=0 the 3Dcoordinates (X) may be simply calculated by finding the unit singularvector corresponding to the smallest singular value of A. This can besolved by using a singular value decomposition (SVD). The matrix A maybe expressed as:

$\begin{matrix}{A = \begin{bmatrix}{{xp}^{3T} - p^{1\; T}} \\{{yp}^{3\; T} - p^{2\; T}} \\{{x^{\prime}p^{{\prime 3}\; T}} - p^{{\prime 1}\; T}} \\{{y^{\prime}p^{{\prime 3}\; T}} - p^{{\prime 2}\; T}}\end{bmatrix}} & (7)\end{matrix}$p^(iT) and p′^(iT) represent the i-th rows of P and P′, respectively,and [x, y]^(T) and [x′, y′,]^(T) represent the image coordinates of thepoint correspondences. For 3D reconstruction, it is preferred not to usea complex optimization algorithm such as a bundle adjustment because thecomputational cost would be too high.2. Feature Selection and 3D Structure Mosaicking2.1 Degradation of 3D Structure Near the Epipole

When reconstructing 3D structures in general, heavy degradation canappear near the epipole. This is because triangulation has to beperformed at a small angle, since the accuracy of the 3D coordinates isdegraded because of the relatively high point detection and imagequantization errors. This can be shown as a rank deficiency of thematrix A in Eq. (7). If the projection matrices of the two cameras arerepresented by P and P′, respectively, they can be written as:P=K[I|0]=[K|0]P′=K[R|t]=[KR|e]  (8)K and I represent a 3×3 camera intrinsic parameter matrix and a 3×3identity matrix, respectively, and R and t represent a 3×3 rotationmatrix and a 3×1 translation vector. e is the epipole. Since the lastcolumn of P′ represents the coordinates of the epipole, the last columnof A becomes closer to zero when the point correspondence nears theepipole.

Even though this problem is very serious in 3D reconstruction, it hasnot really been dealt with in the prior arts because of two reasons.First, the epipole is not located inside the image in many applicationsbecause of camera configurations. This happens when the 3D structuresare reconstructed by using a stereo camera or a single moving camerawhose translation in the optical axis is not more than the translationsin the other axes. Second, the epipole is located inside the image butit is not on the target objects. This happens when a mobile robot with asingle forward (or backward) looking camera moves along a road orcorridor. In this case, the epipole is located inside the image but itis usually on objects far from the camera, so the region around theepipole is not interesting.

In the method of the present disclosure, the translation in the opticalaxis is quite dominant. So, the epipole is always located inside theimage. Also, the epipole is usually located on an adjacent vehicle whichis the present target object used for locating free parking spaces. FIG.3 shows the epipole location in a typical parking situation. As shown inthis figure, the epipole is usually located on the image of an adjacentvehicle due to the motion characteristics of the automobile rearviewcamera.

For this reason, the 3D structure of the adjacent vehicle is erroneouslyreconstructed in the present disclosure. FIGS. 4A, B show the locationof the epipole in the last frame of the image sequence and thereconstructed rearview structure. The typical location of the epipole isdepicted in FIG. 4A while FIG. 4B depicts the structure as seen from thetop after removing the points near the ground plane. In FIG. 4A, the 3Dpoints near the epipole on the adjacent vehicle appear quite erroneous,so the detection results are degraded. To solve this problem,de-rotation-based feature selection and 3D structure mosaicking areused.

2.2 De-Rotation-Based Feature Selection and 3D Structure Mosaicking

To solve the problem of the epipole and obtain a precise 3D rearviewstructure, the present disclosure uses a two-step method. In the firststep, the rotational effect from the optical flow is eliminated and thetranslational effect is retained. Since the optical flow length in apure translation is proportional to the 3D structure accuracy, thepresent disclosure simply throws away the point correspondences whoseoptical flow lengths are shorter than the threshold. This prevents the3D structure from including erroneously reconstructed points. Foreliminating the rotational effect of the optical flow, a conjugaterotation homography may be used. If x and x′ were the images of a 3Dpoint X before and after the pure rotation:x=K[I|0]Xx′=K[R|0]X=KRK ⁻¹ x  (9)so that x′=Hx with H=KRK⁻¹. By using this homography, the presentdisclosure eliminates the rotational effect and relates the pointcorrespondences by a pure translation. FIGS. 5A, B show thede-rotated-based feature selection procedure. Firstly, the pointcorrespondences in the fisheye images are undistorted and transformed asshown in FIG. 5A. After that, the undistorted point correspondences arede-rotated by using a homography, as shown in FIG. 5B. All the linesjoining the point correspondences point toward the epipole because therotational effect is totally eliminated. In this case, the epipole isknown as the focus of expansion. A thick spread L1 of lines indicate theunreliable point correspondences classified by the de-rotation basedmethod. The unreliable point correspondences include the features nearthe epipole and far from the camera. The threshold for the optical flowlength may be set to 10 pixels.

In the second step, the present disclosure reconstructs several 3Dstructures by using the selected point correspondences and mosaics theminto one structure by estimating the similarity transformation. Thesimilarity transformation parameters may be consisted of R (3×3 rotationmatrix), t (3×1 translation vector), and c (scaling) and theleast-square fitting method may be used with the 3D pointcorrespondences known from the tracking results. Since the reconstructed3D points may be erroneous and include outliers, the RANSAC approach isused for parameter estimation. The least-square fitting method can beexplained as follows. There may be given two sets of pointcorrespondences X_(i) and Y_(i); i=1, 2, . . . , n in the 3D space.X_(i) and Y_(i) may be considered as 3×1 column vectors, and n is equalto or larger than 3. The relationship between X_(i) and Y_(i) can bedescribed as:Y _(i) =cRX _(i) +t  (10)The mean squared error of two sets of points can be written as:

$\begin{matrix}{{{\mathbb{e}}^{2}\left( {R,t,c} \right)} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}{{Y_{i} - \left( {{cRX}_{i} + t} \right)}}^{2}}}} & (11)\end{matrix}$If A and B are the 3×n matrices of {X₁, X₂, . . . , X_(n)} and {Y₁, Y₂,. . . , Y_(n)}, respectively, and if UDV^(T) isan SVD of AB^(T)(UU^(T)=VV^(T)=I, D=diag(d_(i)), d₁≧d₂≧ . . . ≧0), thetransformation parameters which minimize the mean squared error can becalculated by:

$\begin{matrix}{{R = {UV}^{T}},{c = {\frac{1}{\sigma_{X}^{2}}\mspace{14mu}{trace}\mspace{14mu}(D)}},{t = {\mu_{Y} - {{cR}\;\mu_{X}}}}} & (12)\end{matrix}$σ² _(X), μ² _(X), and μ² _(Y) can be defined as:

$\begin{matrix}{{\sigma_{X}^{2} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}{{X_{i} - \mu_{X}}}^{2}}}},{\mu_{X} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}X_{i}}}},{\mu_{Y} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}Y_{i}}}}} & (13)\end{matrix}$

FIGS. 6A, B, C show the reconstructed 3D structures with and without theproposed feature selection and 3D mosaicking method. FIGS. 6Aa-6Ac showsthe key frame images, FIGS. 6Ba-6Bc the reconstructed 3D structureswithout using the disclosed feature selection method and FIGS. 6Ca-6Ccthe reconstructed 3D structures with using the feature selection method.The 3D structures are shown as seen from the top (after removing thepoints near the ground). FIGS. 6Aa-6Ac show the key frame images andtheir epipole locations. It can be seen that the epipoles are located ondifferent positions of the adjacent vehicle. FIG. 4B shows thereconstructed 3D structures of each key frame. The structures near theepipoles are badly reconstructed. However, the erroneously reconstructedpart in one structure is correctly reconstructed in another structure.FIG. 4C shows the reconstructed 3D structures of each key frame afterremoving the corresponding points by using the de-rotation-based method.Most of the erroneous 3D points in FIG. 4B are deleted.

FIGS. 7A, B show the comparison of the 3D structures when using theproposed feature selection and 3D mosaicking method and without usingthe proposed method, respectively.

The point CC indicates the camera center. The degradation of the 3Dstructure near the epipole is solved by the present method.

3. Free Parking Space Detection

3.1 Metric Recovery by Using the Camera Height Ratio

For locating free parking spaces in terms of centimeters, the metricinformation of the 3D structure has to be recovered. This is usuallyachieved by using a known baseline length or prior knowledge of the 3Dstructure. Since the camera height in the real world is known in thepresent disclosure, estimation may be made on the camera height in thereconstructed world and the ratio of the estimated camera height to thereal height is used for metric recovery. The camera height in the realworld may be assumed as fixed in this disclosure. The height sensor canbe used with camera height variations that may occur due to changingcargos or passengers.

To calculate the camera height ratio, the ground plane in thereconstructed world has to be estimated because the camera location isset to the origin. The estimation procedure may consist of three steps:tilting angle compensation, density estimation-based ground planedetection, and 3D plane estimation-based ground plane refinement. Thetilting angle is calculated and the 3D structure is rotated according tothis calculation. This procedure forces the ground plane to be parallelto the XZ-plane. FIG. 8 shows a configuration of the rearview camera. Inthe camera configuration of FIGS. 6A, B, C, the tilting angle (θ) may becalculated by using Eq. (14):

$\begin{matrix}{\theta = {\arctan\left( \frac{e_{x} - y_{0}}{f} \right)}} & (14)\end{matrix}$e_(x) and y₀ are the y-axis coordinates of the epipole and the principalpoint, respectively. f is the focal length of the camera.

The ground plane may be roughly detected by using the density of the 3Dpoints in the Y-axis. Since there is usually only one plane (the groundplane) parallel to the XZ-plane, the density of the Y-axis coordinate ofthe 3D points is at the maximum peak at the location of the groundplane. FIG. 9 shows the density of the Y-axis coordinate of the 3Dpoints. In this figure, the peak location is recognized as the locationof the ground plane and the distance from the peak location to theorigin is recognized as the camera height in the 3D structure.

In the experiments, the correct location and the orientation of theground plane were refined by 3D plane estimation. The 3D points near theroughly estimated ground plane were selected and the RANSAC approach wasused for estimating the 3D plane. The camera height was refined bycalculating the perpendicular distance between the camera center and theground plane. The 3D structure was scaled in centimeters by using thecamera height ratio. After that, the 3D points far from the cameracenter were deleted and the remaining points were rotated according tothe 3D plane orientation to make the ground plane parallel to theXZ-plane. FIGS. 10A, B show the final result of the metric recovery.FIG. 8A represents the resultant metric recovery viewed from the cameraand FIG. 8B is the metric recovery viewed from the top, respectively,where the point at (0, 0, 0) indicates the camera center.

3.2. Free Parking Space Detection

Once the Euclidean 3D structure of the rearview is reconstructed, freeparking spaces have to be detected in the 3D point clouds. For thistask, estimation may be made on the position of the adjacent vehiclesand free parking spaces may be located accordingly. Because positionestimation can be complicated and time-consuming, the present disclosurereduces the dimensions of the structure from 3D to 2D by using thetop-view. The 3D points, whose heights from the ground plane are between30˜160 cm, may be selected and the height information may be removed toreduce the dimensions. FIGS. 11A, B show the dimension reduction resultwhere FIG. 11A is the dimension reduction result and FIG. 11B is theisolated point deletion result. As illustrated, in the dimensionreduction result the outlines of the adjacent vehicle and otherobstacles can be seen. In this figure, there are some isolated pointswhich might have been caused by the 3D reconstruction error, so they aredeleted by counting the number of neighbors defined as:N(x,ε)={y|∥y−x∥≦ε}  (15)x and y represent the 2D points and ε represents the radius. If N(x,ε)were lower than the threshold, x is defined as an isolated point anddeleted. FIG. 11B shows the isolated point deletion result with ε andthe threshold set as 10 cm and 3, respectively. The point at (0, 0)indicates the camera center.

Since all the points in FIGS. 11A, B do not belong to the outermostsurface of the automobile, the outline points are selected by using therelationship between the incoming angle and the distance from the cameracenter. This procedure is performed for better estimation of theposition of the adjacent vehicle. The incoming angle is the anglebetween the horizontal axis and the line joining the camera center and a2-D point.

FIGS. 12A, B show the outline point selection where FIG. 12A re-depicts2D points with the incoming angle and the distance from the cameracenter and FIG. 12B is the result of the outline point selection. FIG.11B may be re-depicted in FIG. 12A by using the incoming angle and thedistance from the camera center. Since the points on the same verticalline may come from the same incoming angle in FIG. 12A, the nearestpoint from the camera center among the points on the same vertical linemay be recognized as the outline point.

If the automobile shape is a rectangle as seen from the top, theposition of the adjacent vehicle can be represented by a corner pointand orientation. Therefore, the corner point and orientation of theadjacent vehicle are estimated and these values may be used to locatefree parking spaces. Since the reconstructed structure is noisy andincludes not only adjacent vehicles but also other obstacles, it ispreferable to use a projection-based method. This method rotates the 2Dpoints and projects them onto the X-axis and Z-axes. It discovers therotation angle which maximizes the sum of the maximum peak values of thetwo projection results. The rotation angle and the locations of the twomaximum peak values are recognized as the orientation and the cornerpoint, respectively. This method estimates the orientation and thecorner point at the same time and it is robust to noisy data.

However, when using this method, it is impossible to know whether theestimated orientation is longitudinal or lateral. To determine this, itis assumed that a driver would turn right when a free parking space ison the left, and vice versa. Using this assumption, it is possible todetermine whether the estimated orientation is longitudinal or lateral.The turning direction of the automobile may be estimated from therotation matrix extracted from the fundamental matrix.

After estimating the orientation and corner point, the points in thelongitudinal direction of the adjacent vehicle may be selected and usedfor refining the orientation by using RANSAC-based line estimation. Thisprocedure is needed because the lateral side of automobiles is usuallycurved so the longitudinal side gives more precise orientationinformation. The corner point is also refined according to the refinedorientation.

FIG. 13A shows a search range of the other adjacent vehicle and FIG. 13Bshows a free parking space localization. To locate the most appropriatefree parking spaces, other adjacent vehicles located opposite theestimated vehicle are also searched. The search range may be set as FIG.13A by using the estimated corner point and orientation. A circle havinga radius of 150 cm may be set with its center being located 300 cm awayfrom the corner point in the orthogonal direction of the orientation.

If there were point clouds inside the search range, the other vehicle isconsidered to be found and the free parking space is located in themiddle of two vehicles in the lateral direction. The corner points oftwo adjacent vehicles are projected in a longitudinal direction and theouter one may be used to locate free parking spaces. This is describedin FIG. 13B. In this figure, corner point 1 is selected because this isthe outer one. If the other vehicle (corner point 2) were not found, thefree parking space is located beside the detected vehicle with a 50 cminterval in the lateral direction. FIG. 14 shows the final result of thefree parking detection process depicted on the last image sequence. Thewidth and length of the free parking space in FIG. 14 are set as 180 cmand 480 cm, respectively.

4. Experimental Results

The free parking space detection system was tested in 154 differentparking situations. From the database, 53 sequences were taken with thelaser scanner data and 101 sequences were taken without it to analyzethe results in terms of success rate and detection accuracy.

4.1 Comparison of the Reconstructed Rearview Structures

This experiment reconstructed the 3D rearview structures when using andwithout using the disclosed feature selection and 3D mosaicking methodand compared them to the laser scanner data. The angular resolution anddepth resolution of the laser scanner used were 0.125° and 3.9 mm,respectively and the systematic error was ±25 mm. FIG. 15 shows thefisheye camera and the laser scanner mounted on the automobile. Thesetwo sensors were pre-calibrated.

FIGS. 16A, B, C show comparisons of the reconstructed rearviewstructures when using and without using the disclosed method. FIGS.16Aa-16Ab depict the last frames of two image sequences, FIGS. 16Ba-16Bbthe rearview structures reconstructed when using the disclosed method,and FIGS. 16Ca-16Cb the rearview structures reconstructed without usingthe disclosed method.

The reconstructed structures are depicted as seen from the top afterremoving the points near the ground plane. The points on the vehicleindicate the locations of the epipoles. In FIGS. 16Ba-16Bb, 16Ca-16Cb,the points in solid and outlined areas indicate the reconstructed pointsand the laser scanner data, respectively.

By using this comparison, it is possible to obtain three advantages ofthe disclosed method. First, it reduces the number of erroneouslyreconstructed points. FIGS. 16Ca-16Cb show more erroneous points outsidethe ground truth data than FIGS. 16Ba-16Bb because the disclosed methodremoves the point correspondences near the epipole and far from thecamera center. Second, the present disclosure increases the amount ofinformation about adjacent vehicles. The structure in FIGS. 16Ba-16Bb ismore detailed than that in FIGS. 16Ca-16Cb because the density of thepoints on the adjacent vehicle is increased by the mosaicking 3Dstructures. Last, the present disclosure enhances metric recoveryresults. In FIGS. 16Ca-16Cb, the scale of the reconstructed structurediffers from the ground truth, since the disclosed method produces morepoints on the ground plane, so it makes the ground plane estimation moreaccurate.

4.2 Free Parking Space Detection Results

The disclosed system was applied to 154 real parking situations. Theground planes were covered with asphalt, soil, snow, standing water, andparking markers. The automobiles varied in color from dark to bright andthey included sedans, SUVs, trucks, vans and buses. The environmentincluded various types of buildings, vehicles, and trees. FIGS. 17A, B,C, D show six successful examples. FIGS. 17Aa-17Ac, 17Ca-17Cc show thelast frames of the image sequences and FIGS. 17Ba-17Bc, 17Da-17Dc showthe rearview structures corresponding to FIGS. 17Aa-17Ac, 17Ca-17Cc,respectively. In these figures, (0, 0) indicates the camera center.

To decide whether the system succeeded, free parking space on the lastframe of the image sequence was displayed. If it was located inside thefree space between two adjacent vehicles, the result was considered tobe a success. In this way, the system succeeded in 139 situations andfailed in 15 situations, so the detection rate was 90.3%.

FIGS. 18A, B, C, D show four types of failures due to the reflectedsunlight in FIG. 18A, a dark vehicle under a shadowy region in FIG. 18B,a far parking space in FIG. 18C and an uneven ground plane in FIG. 18D.

In FIG. 18A, the sun was strongly reflected on the surface of theadjacent vehicle and the ground plane, so feature point tracking failed.In FIG. 18B, the adjacent vehicle was very dark and it was located in ashadowy region, so few feature points were detected and tracked on thesurface of the vehicle. In FIG. 18C, the free parking space was very farfrom the camera, so the side of the white car at W was more preciselyreconstructed than that of the silver van at S. This caused falsedetection. In FIG. 18D, part of the ground plane on the parking space atP was repaved with asphalt, so the ground plane was not flat. This madethe ground plane estimation erroneous. Out of fifteen failures, threecould be depicted by FIG. 18A, nine could be depicted by FIG. 18B, twocould be depicted by FIG. 18C, and one could be depicted by FIG. 18D.

4.3 Accuracy of Adjacent Vehicle Detection

The disclosed free parking space detection method estimates the cornerpoint and the orientation of the adjacent vehicle and then locates thetarget space accordingly. Since the detection result depends on theestimation of the corner point and orientation, calculation is performedto obtain their errors for accuracy evaluation. The ground truth of thecorner point and the orientation may be obtained by using laser scannerdata. The error of the corner points is the Euclidean distance from theestimated point to the measured point and the error of the orientationis the absolute difference between the estimated angle and the measuredangle. For this evaluation, 47 image sequences and the correspondinglaser scanner data were used. The corner point and the orientation ofthe adjacent vehicle were estimated 10 times for each image sequence.This is because the reconstructed structure can differ slightly everytime due to the parameter estimation results.

FIGS. 19A, B show evaluation results by using laser scanner data whereFIG. 19A shows the corner point error and FIG. 19B shows the orientationerror. In FIGS. 19A, B, the errors of the corner point and theorientation are depicted as histograms. The average and maximum errorsof the corner point were 14.9 cm and 42.7 cm, respectively. The distancebetween the corner point and the camera center was between 281.4 cm and529.2 cm. Since the lateral distance between two adjacent vehicles isapproximately between 280 cm and 300 cm in a usual garage parkingsituation, there is about 50 cm extra room on each side of the vehicle.This means that even the maximum error of the corner point is acceptablefor the free parking space localization. The average and maximum errorsof the orientation may be 1.4° and 7.7°, respectively. This averageerror of the orientation is acceptable but the maximum error of theorientation may be somewhat large. This is because the side surfaces ofautomobiles show few corresponding points due to featurelessness andthis makes the orientation estimation difficult. This evaluation showsthat the disclosed system produces acceptable results for detecting freeparking spaces.

The above exemplary embodiments of the present disclosure have beendescribed for illustrative purposes, and those skilled in the art willappreciate that various modifications, additions and substitutions arepossible, without departing from the substantial characteristics of thedisclosure. Therefore, exemplary embodiments of the present disclosurehave not been described for limiting purposes.

It is to be understood that the present invention is not limited to theembodiments describe above. The scope of the present disclosure is to beinterpreted by the following claims and the entire technical idea withinthe claimed scope is to be interpreted in determining what is includedin the invention.

What is claimed is:
 1. A vehicle free parking space detection apparatuscomprising: a feature points tracker for tracking feature points byreceiving input images from an image captor and acquiring pointcorrespondences through analyzing the input images; a three-dimensionalstructure generator for generating a three-dimensional structure throughthree-dimensionally reconstructing the free parking space by using thefeature points; a three-dimensional mosaic structure generator forgenerating a three-dimensional structure mosaic through reselecting thefeature points to reconstruct the three-dimensional structure by using ade-rotation-based feature selection and mosaic the reconstructedthree-dimensional structure by estimating a similarity transformation; ametric recoverer for acquiring metric information from thethree-dimensional structure mosaic by using an image captor heightratio; and a free parking space detector for detecting the free parkingspace by estimating positions of adjacent vehicles in thethree-dimensional structure mosaic.
 2. The vehicle free parking spacedetection apparatus in claim 1, wherein the feature points trackerdetects the feature points and tracks the feature points through theinput images by using the Lucas-Kanade method.
 3. The vehicle freeparking space detection apparatus in claim 1, wherein thethree-dimensional structure generator selects key frames of the inputimages and estimates motion parameters and then calculates threedimensional coordinates of the feature points by using a lineartriangulation method to generate the three-dimensional structure.
 4. Thevehicle free parking space detection apparatus in claim 1, wherein thefree parking space detector reduces the three dimensional structuremosaic with a top-view, deletes isolated points from neighbors, acquiresoutline points of the adjacent vehicles with an incoming angle and thedistance from the image captor center and estimates a corner point andorientation of the adjacent vehicle, whereby detecting the free parkingspace.
 5. The vehicle free parking space detection apparatus in claim 4,wherein the free parking space detector additionally estimates otheradjacent vehicles located opposite the estimated vehicle by setting acircle with a predetermined radius as the search range of the otheradjacent vehicles with the center of the circle being located apredetermined distance from the corner point in the orthogonal directionof the orientation.
 6. The vehicle free parking space detectionapparatus in claim 1, wherein the metric recoverer acquires the imagecaptor height ratio by determining the image captor height in the threedimensional mosaic structure through a tilting angle compensation, adensity estimation-based ground plane detection, and a three-dimensionalplane estimation-based refinement of the ground plane location andorientation.
 7. A vehicle free parking space detection apparatuscomprising: an image captor for capturing the free parking space togenerate input images; a user interface for displaying the detected freeparking space in an output image; a free parking space detector forreceiving the input images to track feature points by acquiring pointcorrespondences through analyzing the input images, generating athree-dimensional structure through three-dimensionally reconstructingthe free parking space by using the feature points, generating athree-dimensional structure mosaic through reselecting the featurepoints to reconstruct the three-dimensional structure by using ade-rotation-based feature selection and mosaic the reconstructedthree-dimensional structure by estimating a similarity transformation,acquiring metric information from the three-dimensional structure mosaicby using an image captor height ratio, and detecting the free parkingspace by estimating positions of adjacent vehicles in thethree-dimensional structure mosaic to generate the detected free parkingspace; a sensor having a number of sensing units for recognizing drivingconditions of the vehicle to generate and deliver vehicle drivingcondition information; a parking assistance controller for receiving thedriving condition information from the sensor to estimate a position ofthe vehicle, generating a path plan to park the vehicle into the freeparking space by using the detected free parking space delivered, andgenerating and delivering control signals to execute the parking intothe free parking space; an active steering system for steering thevehicle in response to the control signals; and an active breakingsystem for breaking the vehicle in response to the control signals.
 8. Amethod for a vehicle free parking space detection apparatus to detectthe free parking space comprising the steps of: tracking feature pointsby receiving input images from an image captor and acquiring pointcorrespondences through analyzing the input images; generating athree-dimensional structure through three-dimensionally reconstructingthe free parking space by using the feature points; reconstructing thethree-dimensional structure through reselecting the feature points byusing a de-rotation-based feature selection; generating athree-dimensional structure mosaic through mosaicking the reconstructedthree-dimensional structure by estimating a similarity transformation;acquiring metric information from the three-dimensional structure mosaicby using an image captor height ratio; and detecting the free parkingspace by estimating positions of adjacent vehicles in thethree-dimensional structure mosaic.